Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.14.1
Description
jdbc query results exceed native heap when using generous -Xmx settings.
for roughly 800 megabytes of csv/flatfile resultset, arrow is unable to house the contents in RAM long enough to persist to disk, without explicit knowledge beyond unit test sample code.
source:
https://github.com/jnorthrup/jdbc2json/blob/master/src/main/java/com/fnreport/QueryToFeather.kt#L83
Exception in thread "main" org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of size 4 due to memory limit. Current allocation: 2147483646 at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:307) at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:277) at org.apache.arrow.adapter.jdbc.JdbcToArrowUtils.updateVector(JdbcToArrowUtils.java:610) at org.apache.arrow.adapter.jdbc.JdbcToArrowUtils.jdbcToFieldVector(JdbcToArrowUtils.java:462) at org.apache.arrow.adapter.jdbc.JdbcToArrowUtils.jdbcToArrowVectors(JdbcToArrowUtils.java:396) at org.apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(JdbcToArrow.java:225) at org.apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(JdbcToArrow.java:187) at org.apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(JdbcToArrow.java:156) at com.fnreport.QueryToFeather$Companion.go(QueryToFeather.kt:83) at com.fnreport.QueryToFeather$Companion$main$1.invokeSuspend(QueryToFeather.kt:95) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241) at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:270) at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79) at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54) at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source) at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36) at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) at com.fnreport.QueryToFeather$Companion.main(QueryToFeather.kt:93) at com.fnreport.QueryToFeather.main(QueryToFeather.kt)
Attachments
Issue Links
- is related to
-
ARROW-6219 [Java] Add API for JDBC adapter that can convert less then the full result set at a time.
-
- Resolved
-
- links to
